Objective Evaluation of Subjective Decisions

The Dempster-Shafer “theory of evidence” encompasses and extends the Bayes Theorem-based decision making machinery. Dempster-Shafer’s innovation is the introduction of lower and upper bounds, designated “belief” and “plausibility”, that are attached to probability estimates. The Dempster-Shafer algebra provides for propagation of and reasoning about these quantities according to an algebra whose outcome phenomenologically mimics human decision making in many contexts that are laden with quantitative uncertainty. The approach’s decisions thus seem to be subjective, i.e., the product of a sentient mind, vs. objective, i.e., the mechanical outcome of an immutable algorithm. In this paper we address the “objective evaluation of subjective decisions” in particular with the Dempster-Shafer sort of “subjective” decision making algorithm in mind. As an initial baseline approach, we examine the “receiver operating characteristic” (ROC) graph. We regard this as a first step towards identifying in advance circumstances under which Dempster-Shafer-like approaches should and should not be expected to deliver results that pass the human “sanity test”.